Add the FTRL (Follow The Regularized Leader) optimizer. #268
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![copybara-service[bot]](https://avatars.githubusercontent.com/in/44061?s=60&v=4){kind=small}
copybara-service
bot
force-pushed
the
test_764794731
branch
from
4f30082 to
2fee215
Compare
copybara-service
bot
changed the title
copybara-service
bot
force-pushed
the
test_764794731
branch
4 times, most recently
from
63f5235 to
e46ff56
Compare
copybara-service
bot
force-pushed
the
test_764794731
branch
5 times, most recently
from
214e140 to
4d2fa70
Compare
This implementation is based on the FTRL algorithm, [McMahan et al., 2013](https://research.google.com/pubs/archive/41159.pdf). Features / Params in `FTRLOptimizerSpec` (as used in the primitive): - **learning_rate**: The base learning rate. - **learning_rate_power**: Controls the per-coordinate learning rate decay (typically -0.5). - **l1_regularization_strength**: Applies L1 regularization, which can lead to sparsity in the model weights. - **l2_regularization_strength**: Applies L2 regularization. - **beta**: An additional smoothing term.. - **multiply_linear_by_learning_rate**: Boolean flag; if true, the linear term update incorporates the `learning_rate` differently. The optimizer maintains two slot variables for each trainable embedding parameter: - **accumulator**: Stores the sum of squared gradients, used to adapt the learning rate on a per-coordinate basis. - **linear**: Stores a linear combination related to the gradients, which is central to the FTRL weight update rule. PiperOrigin-RevId: 769851368
copybara-service
bot
force-pushed
the
test_764794731
branch
from
4d2fa70 to
c02c8ea
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add the FTRL (Follow The Regularized Leader) optimizer.
This implementation is based on the FTRL algorithm, McMahan et al., 2013.
Features / Params in
FTRLOptimizerSpec(as used in the primitive):learning_ratedifferently.The optimizer maintains two slot variables for each trainable embedding parameter: